Controlling Power Consumption Based on User Gaze

ABSTRACT

The determination of a gaze direction or field of view of a user with respect to a computing device can control aspects of the device, such as to reduce power or resource consumption. A computing device can include an imaging element and software for locating aspects of a user&#39;s facial features relative to the device, such that an orientation of the user&#39;s features relative to the device can be determined. Various actions on the device can be executed based at least in part upon a determined gaze direction of the user. In some embodiments, a display element of the device can turn off, and one or more inputs can be disabled, when the user is not looking at the device.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/718,788, entitled “ACTION-BASED DEVICE CONTROL” filed Mar. 5, 2010,and issuing as U.S. Pat. No. 8,913,004, which is incorporated herein byreference for all purposes.

BACKGROUND

As the variety of available computing devices increases, and as the sizeof many of these devices decreases, there comes a need to adapt the waysin which users interact with these computing devices. Peopleincreasingly utilize portable devices such as cellular phones, personaldigital assistants (PDAs) and other electronic devices, which presentdifferent challenges than conventional stationary devices such asdesktop computers. For example, portable devices typically are poweredby a battery that has a limited operational time. For devices with largedisplay elements or processor-intensive functionality, for example,usage can drain the batteries relatively quickly. Even for devices thatcan be plugged into a wall socket or powered using other conventionalmechanisms, however, it still can be desirable to reduce the powerconsumption of these devices.

Certain conventional devices utilize various inactivity-basedapproaches, such as turning off a display of a cellular phone if a userdoes not interact with an input element such as a keypad,touch-sensitive display or roller ball for a determined period of time.Such an approach is not optimal, as the user can become frustrated ifthe display shuts off too quickly or while the user is viewing contenton the device.

Further, users of devices such as smart devices and cellular phonesoften trigger undesired actions by inadvertently causing specific keysor other input mechanisms to be activated when placing the devices intoa pocket, purse, backpack, etc. Any particular movement can cause anaction to occur, such as a phone to dial an emergency number or a smartdevice to continually illuminate the display element. Thus, in additionto reducing power consumption, it can be desirable to limit thefunctionality of various devices in certain situations.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example of a portable device including an elementoperable to image the facial region of a user in accordance with oneembodiment;

FIG. 2 illustrates components of a controllable device that can be usedin accordance with one embodiment;

FIGS. 3( a)-3(c) illustrate approaches to altering the state of adisplay element based on the gaze direction of a user that can be usedin accordance with one embodiment;

FIGS. 4( a)-4(b) illustrate approaches to automatically calibrate gazedirection determinations with respect to a computing device inaccordance with one embodiment;

FIG. 5 illustrates an example process for adjusting a display element inresponse to detected changes in gaze direction that can be used inaccordance with various embodiments;

FIG. 6 illustrates an example of a display that can be generated toassist in recalibrating gaze detection in accordance with oneembodiment;

FIGS. 7( a)-7(b) illustrate analysis of facial features of a user inaccordance with various embodiments;

FIGS. 8( a)-8(c) illustrate an example of capturing eye movement of auser as input in accordance with one embodiment;

FIGS. 9( a)-9(c) illustrate an approach to determining retina locationfrom a pair of images that can be used in accordance with oneembodiment;

FIG. 10 illustrates an example computing device that can be used inaccordance with various embodiments;

FIG. 11 illustrates example actions that can be taken in response toinput combinations with respect to a computing device that can be usedin accordance with various embodiments;

FIG. 12 illustrates an example of a display that can be presented to auser to customize device actions in accordance with one embodiment;

FIG. 13 illustrates an example process for taking determined actions ona computing device in response to changes in gaze direction or devicestate in accordance with one embodiment;

FIG. 14 illustrates an example of a display including icons forprocesses that can be at least partially executed based upon user gazedirection in accordance with one embodiment;

FIG. 15 illustrates an example of a multi-dimensional display that canbe rendered based at least in part upon the viewing location of a userin accordance with one embodiment;

FIG. 16 illustrates an example of a display that can be rendered basedat least in part upon the viewing location of a user in accordance withone embodiment; and

FIG. 17 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments overcome manyof the aforementioned deficiencies experienced in conventionalapproaches to controlling electronic devices. In one embodiment, animaging element captures an image of a device user and determines whichdirection the user is looking relative to the device (referred to hereinas “gaze direction”). The determined gaze direction of the user is usedas input to control various aspects of the device, such as power andfunctionality. For example, if it is determined that the user's gazedirection is not toward the device, the display element may shut off andvarious device functions may be shut off, thereby reducing powerconsumption.

In addition to utilizing an imaging element to determine a user's gazedirection relative to the device, some embodiments described herein mayalso include additional elements, such as touch-sensitive elements,orientation determining elements and/or light sensing elements, that canassist in applying context to the user's determined gaze direction,which can assist in determining controllable actions to take withrespect to the device. For example, the device may perform a firstaction when the user is not looking at the device but is holding thedevice and may perform a selected action when the user is not lookingat, nor holding, the device.

In various embodiments, a computing device can include at least onedisplay element. A user of the computing device will typically be facingthe display element of the computing device during use, at least forspecific applications or uses. The computing device can include, or bein communication with, at least one imaging element, such as a camera,detector or sensor, that can be positioned with respect to the devicesuch that the imaging element is able to obtain images of at least aportion of the user while the user is operating the device. In oneembodiment, a high speed, high resolution camera that is capable ofcapturing a large number of images in a small amount of time isutilized. The device can analyze image information obtained from theimaging element and determine at least one relative location ororientation of a user with respect to the device, as well as a gazedirection or field of view of the user. The image processing algorithmcan also determine a change in the gaze direction with respect to thedevice.

In some embodiments, an imaging element of the computing device caninclude at least one infrared emitter and at least one infrared detectorthat can be positioned with respect to the device such that the deviceis able to obtain images of infrared radiation reflected from the user'seyes. The device can analyze image information relating to the relativesize, separation and position of the user's pupils, for example, and candetermine a gaze direction or field of view of the user with respect tothe device. In one embodiment, the device utilizes continuous orperiodic infrared illumination to monitor changes in gaze direction withrespect to the device.

In various embodiments, an image processing algorithm is used todetermine and monitor a user's gaze direction relative to the device ina series of obtained images and determine/identify the relative positionand/or field of view of the user's eyes with respect to the displayelement and/or imaging element. Based on the determined gaze directionand/or field of view of the user's eyes with respect to the displayelement and/or imaging element, the algorithm can determine at leastwhether the user is looking substantially at, or away from, the displayelement. In another embodiment, the algorithm may further determine,based on the gaze direction and the field of view of the user's eyeswith respect to the display element and/or imaging element, where theuser is looking with respect to the display element, such as at aparticular icon or even focused about a particular pixel or otherdisplay element. By tracking changes in the field of view/relativeposition of the user's eyes, the algorithm can determine at any timewhether the user is looking at the device and, if so, where on thedevice the user is looking.

As illustrated in the example above, the functionality of the device canbe altered based at least in part on the determined field of view,viewing position of the user, a change in gaze direction, etc. In someembodiments, the display element can have different brightness values,or even shut off, based at least in part upon whether the user islooking at, toward or away from the display element and/or device. Thatis, in such embodiments, the imaging element is used to provide input tocontrol aspects of the device.

In other embodiments, various functionality or inputs of a device can beenabled/disabled, turned off or otherwise modified based at least inpart upon the determined gaze direction of the user. For example, if itis determined that a user is not looking at the display element of thedevice but attempting to type a message using a touch-sensitive displayelement keyboard, instead of a conventional keyboard, the keyboardinputs may be modified to improve the user's typing ability. Forexample, the device may audibly present each key when selected by theuser. In another example, the device may audibly present each key as theuser positions their finger over/on the key prior to selection. Inaddition to audible presentation of key selections, the device may shutoff the display element to conserve power.

Any number of device features and/or inputs may be controlled inaccordance with the various embodiments described herein. For example,if it is determined that the user's gaze direction is not toward thedevice, or that the imaging element did not capture an image thatincludes the user's face, the device may disable inputs such asone-touch dialing in order to prevent inadvertent selection of such aninput. In at least some embodiments, the user can configure whichfunctions are enabled/disabled as well as define which functions requirethe user to at least look at the display element while performingcertain actions.

Although the embodiments described above generally relate to managingpower consumption and/or managing input to a device, embodiments canalso be used to predict actions by a user in order to pro-activelyinitiate certain processes that can improve response time as well asprevent problems such as processor spikes or memory overruns. Forexample, a view monitoring algorithm can determine when a user islooking at a certain icon on a display element, such as an icon that isused to access documents from a hard disk. If the user looks at the iconfor a minimum period of time, such as a fraction of a second, the devicecould cause the hard drive to spin up, which can reduce the amount oftime needed to actually access a file subsequently selected by the user.If the user's eyes come to rest on an Internet icon, for example, thedevice could preemptively submit a request according to the home page ofthe browsing application on that device, so that the content loadingprocess can begin before the user actually opens the browser. If a userlooks toward an element to shut down the device, the device can beginpowering down or closing unimportant components, applications and/orservices, for example, which can reduce the length of the shut downprocess and conserve resources. If an animated advertisement isdisplayed on the display element, the animation may only be active whenthe device detects that that user is looking at the advertisement.Various other examples are described below with respect to the variousembodiments.

Resources can be further conserved in various embodiments by reducing oreliminating unnecessary processing of instructions. For example, a usermight be playing a game or viewing a detailed three-dimensional image ona device with a large display element. In at least some cases, theuser's primary focus will he limited to a portion of that displayelement. For objects outside the primary focus region, the user will notbe able to tell a significant difference between highly renderedcharacters or features and characters or features rendered with lessdetail. Various embodiments can track the user's gaze and can rendercertain portions of the display element in more or less detail based atleast in part upon the determined gaze direction. By not rendering theentire display in high resolution, for example, the device can reducethe amount of processing power needed or can focus the processing poweron rendering that portion of the display that will be most noticeable tothe user. For three-dimensional displays, or other graphical interfaces,similar approaches can be used to render the information currently beingviewed in a higher level of detail than other information containedwithin the area of the display element but not currently viewed, or atleast focused on, by the user. In other embodiments, in addition to notrendering the entire display in high resolution, the system user canalso interact with objects in that portion of the screen by trackinggaze direction to a specific portion of the screen. For example, if theuser is playing a game, the user can interact with objects in theprimary focus area by looking directly at those objects, such asshooting at enemies, picking up items, talking to other characters, etc.

In an example where a user is browsing items offered through anelectronic marketplace, such as via a website of an e-commerce retailer,the user might view a number of items corresponding to a search requestresponse or category of items. If the items are displayed in athree-dimensional grouping by category, for example, the device canrender the items within the user's gaze differently than items outsidethe user's gaze.

In various embodiments, one or more additional inputs can be used tointerpret, or enhance, information relating to the field of view or gazedirection of a user. For example, the device might include alevel-determining element or accelerometer capable of determining anorientation, as well as movement, of the device. If it is determinedthat the user is not gazing at the device because the user moved thedevice, instead of the user looking away from the device, then thedevice might take a different action with respect to the displayelement, inputs, etc.

In other embodiments, the device can include touch-sensitive elements,over at least a portion of the device, such that the device candetermine whether a user is currently holding the device. In suchembodiments, the device can function differently if the user is holdingthe device but not looking directly at the display element, such as whenthe user is typing while looking elsewhere, than when the user is notlooking at, or holding, the device.

In other embodiments, the device can include an element, such as aconventional light-sensing element, that is able to determine whetherthe device has been placed into a dark area, such as a pocket orbriefcase. For example, if the user is not looking at the device and thelight-sensing element detects that the device moves from a place withambient lighting to an area where there is little to no light, thedisplay intensity can be altered.

As discussed, many of the actions of a user with respect to a device canbe detected using an image-based approach, which is able to determine agaze direction, or field of view, of a user with respect to the device.FIG. 1 illustrates a configuration 100 in accordance with one embodimentwherein a portion of a user 106 operating a portable computing device102 is imaged by an imaging element 104 of the portable device 102. Itshould be understood that the relative orientation of the user and thedevice shown in the figure is shifted for purposes of explanation, andthat the user generally faces the side (e.g., front, back or other side)of the device that includes a display element and the imaging element104. The imaging element 104 may be, for example, a camera, acharge-coupled device (CCD), a motion detection sensor or an infraredsensor, among many other possibilities. It should also be understoodthat, while a portable computing device 102 is depicted in this example,the portable device could be any appropriate device able to receive andprocess input commands, such as a personal computer, laptop computer,notebook, netbook, television set top box, cellular phone, PDA,electronic book reading device, video game system or portable mediaplayer, among others.

In this configuration, the imaging element 104 is on the same generalside of the portable device 102 as a display element such that when auser is viewing information in the display element, the imaging elementhas a viewable area 108 that, according to this example, includes theface of the user 106. While in some embodiments the imaging element isfixed relative to the device, in other embodiments the imaging elementcan be operable to track the position of the user, such as by rotatingthe imaging element or an optical element (e.g. lens, mirror, etc.) thatdirects light to the imaging element. Although embodiments describedherein use examples of the viewable area including the face of the user,the viewable area may include other portions of the body such as arms,legs and hips, among other possibilities. In either case, the viewablearea 108 of the imaging element can be configured to obtain imageinformation corresponding to at least a portion of a user operating thedevice. If the imaging element is continually (or at least substantiallycontinually) capturing or otherwise obtaining image information, thenany movement of the user 106 relative to the device 102 (throughmovement of the user, the device or a combination thereof) can cause aposition or orientation of at least one aspect of that user (e.g., faceor eye location) within the viewable area 108 to change. If the deviceincludes software and/or hardware that is able to locate at least onesuch feature of the user that can be consistently determined, then thedevice can analyze the image information to manage functionality of thedevice over a period of time and utilize that relative motion as asource of input.

For example, a user can glance away from the portable device. Suchmotion can be detected and analyzed by the imaging element (e.g.,camera) as the position of the user's eyes in the viewable area 108 willmove in the images or may no longer be contained within the area of thecaptured images. Further, aspects such as the imaged shape, size andseparation of the user's eyes also can change as discussed elsewhereherein.

A change in eye position in the viewable area could also be accomplishedby moving the device up and down while the user remains still, as wellas through other such motions. In some embodiments, the device is ableto distinguish between movement of the user and movement of the deviceby detecting movement of a background or other aspect of the images orby analyzing the separation, shape or size of various features. Thus, inembodiments described anywhere in this description that use the imagingelement to determine an orientation of the device relative to its user,the tracking of information such as gaze direction can account formovement of the device, such that a change in gaze direction due tomovement of the device relative to the user can be accounted for withoutinadvertently registering the change in gaze direction as a new inputvalue, etc.

FIG. 2 illustrates a set of basic components of an example computingdevice 200 such as the device 102 described with respect to FIG. 1. Inthis example, the device includes a processor 202 for executinginstructions that can be stored in a memory device or element 204. Asknown in the art, the device can include many types of memory, datastorage or computer-readable media, such as a first data storage forprogram instructions for execution by the processor 202, a separatestorage for images or data, a removable memory for sharing informationwith other devices, etc. The device typically will include some type ofdisplay element 206, such as a liquid crystal display (LCD), althoughdevices such as portable media players might convey information viaother means, such as through audio speakers. As discussed, the device inmany embodiments will include at least one imaging element 208 such as acamera that is able to image a facial region of a user. The imagingelement can include any appropriate technology, such as a CCD imagingelement having a sufficient resolution, focal range and viewable area tocapture an image of the user when the user is operating the device.Methods for capturing images using an imaging element with a computingdevice are well known in the art and will not be discussed herein indetail. It should be understood that image capture can be performedusing a single image, multiple images, periodic imaging, continuousimage capturing, image streaming, etc. Further, a device can include theability to start and/or stop image capture, such as when receiving acommand from a user, application or other device.

In some embodiments, the device can have sufficient processingcapability, and the imaging element and associated analyticalalgorithm(s) may be sensitive enough to distinguish between the motionof the device, motion of a user's head, motion of the user's eyes andother such motions, based on the captured images alone. In otherembodiments, such as where it may be desirable for the process toutilize a fairly simple imaging element and analysis approach, it can bedesirable to include at least one orientation determining element 210that is able to determine a current orientation of the device 200. Inone example, the at least one orientation determining element is atleast one single- or multi-axis accelerometer that is able to detectfactors such as three-dimensional position of the device and themagnitude and direction of movement of the device, as well as vibration,shock, etc. Methods for using elements such as accelerometers todetermine orientation or movement of a device are also known in the artand will not be discussed herein in detail. Other elements for detectingorientation and/or movement can be used as well within the scope ofvarious embodiments for use as the orientation determining element. Whenthe input from an accelerometer or similar element is used along withthe input from the camera, the relative movement can be more accuratelyinterpreted, allowing for a more precise input and/or a less compleximage analysis algorithm.

In some embodiments, the device can include at least one additionalinput device 212 able to receive conventional input from a user. Thisconventional input can include, for example, a push button, touch pad,touch-sensitive element used with a display, wheel, joystick, keyboard,mouse, keypad or any other such device or element whereby a user caninput a command to the device. Some devices also can include amicrophone or other audio capture element that accepts voice or otheraudio commands. For example, a device might not include any buttons atall, but might be controlled only through a combination of visual andaudio commands, such that a user can control the device without havingto be in contact with the device. As will be discussed later herein,functionality of these additional input devices can also be adjusted orcontrolled based at least in part upon the determined gaze direction ofa user or other such information.

FIGS. 3( a)-3(c) illustrate an example situation 300 wherein adetermined gaze direction of the user is used to control an aspect of adevice in accordance with at least one embodiment. In this example, anaspect of the display element is adjusted based on whether the user islooking at the display element 310 or elsewhere. In FIG. 3( a), the user302 is positioned in front of an imaging element 308 of a computingdevice 304. The imaging element 308 is able to capture images of theuser 302 during operation of the device, and instructions on the deviceenable at least one processor to determine a gaze direction 306 of theuser with respect to the device 304. Based at least in part upon thedetermined gaze direction, the processor can manage an aspect of thedisplay element 310, such as to control a brightness of an LCD displayor the display of characters on an electronic paper display.

In FIG. 3( a), the device determines that the gaze direction 306 of theuser is such that the user is looking at the display element 310 of thedevice 304. In this embodiment, the display element 310 is activated sothat the brightness, contrast and/or other such elements enable the userto readily view the displayed content. The device can track changes inthe gaze direction 306 of the user while the user interacts with thedevice 304, such as when the user adjusts gaze direction 306 to readinformation on the display element 310, looks toward a keypad (notshown) to type information or performs another action. In someembodiments, as long as the user is looking at the display element 310or at least substantially looking at the device 304, the display elementcan maintain its active settings.

In FIG. 3( b), the device 304 detects that the user 302 is substantiallyfacing the device but is glancing away from the display element 310and/or device. In some embodiments, the display element can be shut offwhenever the user is not looking at the display element and/or device.When the user is facing the device, however, the frequent turning on andoff of a display element (particularly a backlit device such as an LCDdisplay) each time the user looks slightly away can in some cases bedistracting or even annoying to the user and can potentially degrade theuser experience. In this embodiment, if a user is still facing thedevice such that the display element is still likely in at least theperipheral view of the user, the display element may enter anintermediate state, or other such state, wherein at least one aspect ofthe display element is adjusted to reduce the power consumptionattributable to the display element.

In some embodiments, a brightness or contrast level of the displayclement can be adjusted to a lower level when the user is facing thedisplay element but glancing away or even when the user faces away fromthe display element for a short period of time, such as under a selectedthreshold amount of time (e.g., less than a second or two). In otherembodiments where aspects of the display element are sufficientlyvariable, the brightness, contrast and/or other aspects of the displayelement can be adjusted by an amount that is proportional to thedistance of the gaze direction 306 from the display element and/ordevice.

For example, as the gaze direction of the user moves away from thedisplay element, the brightness level of the display element can beginto decrease. As the user looks further away from the display element,the brightness level can continue to decrease. At some point, such as athreshold distance, the user's gaze direction can be sufficiently faraway from the display element that the display element will drop to aminimum brightness value, shut off, go into a sleep mode or perform asimilar action that minimizes or eliminates the power consumptionattributable to the display element when the user is looking at least athreshold distance away. In embodiments where each pixel of a displayelement is illuminated individually, the device can reduce theresolution of the display element, or dim at least some of the pixels,in order to save power. As described above, the threshold and functionscan be selected/defined by each user.

In another embodiment, content may be also be modified when the user isfacing the display element but glancing away or even when the user facesaway from the display element for a short period of time, such as undera selected threshold amount of time (e.g., less than a second or two).For example, if the display element is rendering video, animation or anyother form of moving/multimedia content, as the gaze direction of theuser moves away from the display element, the moving or animated contenton the display element may be paused until the user returns their gazedirection to the display element. Pausing the rendering of movingcontent saves power and enables the user to glance away from the displayelement and resume viewing where they were once their gaze returns tothe display element.

In FIG. 3( c), the device determines that the user is not facing thedevice 304, or at least is looking a threshold distance away. In somecases the user's eyes will not appear (or be detected) in the imagescaptured by the device. In other cases, the user's gaze direction can bedetermined to be sufficiently away from the device to adjust the displayelement to a reduced state, such as to power off the display element. Asshould be understood, however, as the user's gaze direction returnstowards the display element, the display element can either power upcompletely or can adjust aspects (e.g., increase brightness or contrast)proportionally as the gaze direction returns toward the display element.

In order to properly adjust the display element parameters based uponuser gaze direction, it can be necessary in at least sonic embodimentsto calibrate the device with respect to the user. For example, userswill have different eye sizes, separations and other physicaldifferences. Further, users can hold the same device at differentdistances and at different orientations with respect to the user. Therelative orientation also can depend at least in part upon the taskbeing performed by the device, as a user typically will hold or positiondifferently when viewing a movie than when texting or working on aspreadsheet, for example.

In one embodiment, a user can be asked to go through a calibrationprocedure each time the user powers on the device, opens an applicationon the device or performs a similar action. For example, a user could beasked to look at each of a number of targets on a display element, suchas at each corner, whereby the device can detect relative positions ofthe pupils to the eyes of the user for different locations on thedisplay. Once information is obtained for specific gaze directions,other gaze directions can be interpolated using the initial calibrationinformation. If the device is not tracking the gaze direction properly,the user can have the option of recalibrating such as by selecting arecalibration option on the device.

For certain embodiments, users or applications, however, such acalibration procedure might not he desirable or could degrade the userexperience, particularly where the user has to recalibrate the deviceeach time the user activates, or performs a specific action with respectto, the device. Accordingly, FIGS. 4( a)-4(b) illustrate an examplecalibration procedure 400 that can be used in accordance with variousembodiments. In this embodiment, it can be assumed that a user turningon a device will be glancing at the display element within an initialtime period of turning on the device. In other embodiments, past ordefault calibration information can be used as a starting point as soonas a user's eyes are detected by the camera, and it is determined thatthe user is likely looking at the display element. In some embodiments,the device associates and stores the calibration information with arecognition of the user that is generated from an image of the user.Thus, once calibrated for a user, when the user interacts with thedevice, the device will recognize the user and utilize the storedcalibration information. This embodiment provides the ability formultiple users to interact with the device, each having differentsettings and calibration information.

FIG. 4( a) illustrates three different zones, a first zone 402, with aboundary 403, where the user is determined to be gazing or lookingsubstantially at the display element, a second zone 404 where the useris determined to be gazing or looking away from the display element butstill substantially facing the device and a third zone 406 where theuser is determined to be substantially gazing or facing away from thedevice, such as may be defined by a maximum gaze direction boundary, orsecond boundary 405, at the outer edge of the second zone. It should beunderstood that additional or fewer zones could be used in otherembodiments, such as two zones which cause the display element to beeither on or off

Initially, when a user powers up the device or performs a similaraction, the device can at some time detect the user's eyes and candetermine that the user is likely looking toward the device. The devicecan then begin tracking the user's gaze with respect to the device.Because the device has not calibrated the user's relative position,features, etc., the device can allow for a relatively large range of eyemovement before adjusting the display element settings or other inputs.For example, the first boundary 403 and second boundary 405 used toadjust parameters of the display element can be set to be sufficientlyfar from the device such that the device is unlikely to improperlyinterpret the gaze direction of the user. Since the eye separation, sizeand shape of a user will fall within a given range, the device can use adefault zone boundary (or threshold, etc.) that will be used to adjustdisplay clement parameters and/or other inputs such that no typicalhuman would be viewing the display element if determined to be lookingbeyond the first boundary 403.

As the device tracks the movement of the user's eyes over time, thedevice can begin to adjust (e.g., shrink) the size of the zones bydecreasing the first boundary 403 and second boundary 405 or reducingthe distance from the display element at which the user's gaze directioncan cause the parameters of the display element to be adjusted. Forexample, a user interacting with the device will look primarily at thedevice display element as well as other inputs or components of thedevice. By monitoring the view position of the user interacting with thedevice, the device can approximate the outer edges of the device withrespect to the user's gaze position. For example, if the user is readinga book on an electronic book reader, the user will scan almost theentire display element with a significant number of the pages. Bytracking the user's gaze position for each page, the device canautomatically calibrate the user's gaze direction and can update thatcalibration information as it changes, as may be due to the usershifting position, etc. Further, if a user is typing on atouch-sensitive display element, then the device can reasonablyinterpret the user's gaze to be looking at the characters and/or textbeing entered. Various other activities can be correlated with gazedirection or viewing location as should be apparent in light of theteachings and suggestions contained herein.

FIG. 4( b) illustrates a second view of the calibration zones aftermonitoring the user's gaze direction for a period of time. In thisexample, it can be seen that the first boundary 403 surrounding thefirst zone 402 where the user is determined to be substantially lookingat the display element has been reduced to approximately the size of thedisplay element, with a little extra expansion to account for smallvariations, movement, drift, etc. The second boundary 405, surroundingthe second zone 404, has also been reduced such that the display elementcan adjust and/or turn off more quickly or respond better to smallerdetected movements. It should also be understood that other aspects ofthe zones can change as the calibration of the device becomes moreaccurate, as the zones can change in any of shape, number, sizecorresponding action or other such aspects, within the scope of thevarious embodiments. Similarly, the calibration can learn habits orcharacteristics of the user. For example, when a certain user is typinga response to a difficult question from his manager, the user may alwayslook up and to the right before sending the response. Rather thandimming the screen when this user looks up and to the right, the devicedisplay would remain in the active state for the user's selected periodof time.

Once a default calibration has been selected, or calibration informationhas been obtained, the gaze direction of a user can be used to controlthe display element or other functions of the device, as discussedherein. As an example, FIG. 5 illustrates a process 500 that can be usedto turn on and turn off a display element of a device in accordance withvarious embodiments. In this process, a facial recognition mode (orsimilar mode) of the device is activated 502 using approaches such asthose discussed herein. In some instances this might occur automaticallyat startup, and in other instances this might require first receiving anactivation request on behalf of the user. An imaging element of thedevice begins to obtain image information corresponding to the viewablearea of the imaging element 504, and the images are analyzed to locateat least one desired facial feature of the user 506. In some cases, thisprocess might require analyzing one or more images, or a stream ofimages, to locate features such as the eyes of the user. Onceappropriate facial features of the user are located within an acceptabledegree of certainty, various aspects of those facial features aredetermined 508. For example, if the features include a user's eyes, analgorithm can determine the dimensions of each eye, the separation, therelative position with respect to the face, distance from the nose orother such aspects, such that a change in orientation can be detected.In other embodiments, this can further include information such asdimensions of the user's pupils and relative positions with respect tothe eye positions, such that a change in gaze direction can bedetermined. Based at least on the determined aspects, an initial gazedirection or view location of the user with respect to the device isdetermined 510. In some embodiments, a device might require a user toface the device during an initial calibration period. In other devices,such as where user information is stored, the device can capture asingle image and analyze the information to determine a relativeorientation by comparing measured aspects from the image with storedaspect information for the user.

Once the initial field of view is determined, image information can beanalyzed continually (or periodically) in order to monitor variations inthe determined aspects of the facial features. For example, eachcaptured image (or a subset thereof) can be analyzed to determinewhether the user is looking at the display element 512, such that theuser's view location is substantially within a region (or other sucharea) of the display element of the device. If the user is looking atthe display element, a determination can be made as to whether thedisplay element is currently on, illuminated, activated, etc. 514. Ifthe display element is not on, the display element can be turned on orotherwise activated 516 and monitoring can continue. In some cases,state information can be stored for the display device such that thedevice does not need to continually determine whether the displayelement is activated. Various other such approaches can be used as wellwithin the scope of the various embodiments.

If it is determined that the user is not looking at the display element,a determination can be made as to whether the display element iscurrently in an off (or other less than fully active) state 518. If thedisplay element is off, monitoring can continue. In some embodiments, anextended period of inactivity can cause other actions, such as thepowering down of the entire device. If the display element is on, thedisplay element can be turned off 520 or otherwise moved to a state thatconserves power and monitoring can continue. As discussed, orientationinformation from an orientation detection element such as anaccelerometer can also be used to make determinations as to user gazedirection or other such aspects.

In order to accurately determine field of view over time, such thatactions involving the display element and other elements/functions canbe performed at appropriate times, it can be desirable to updatecalibration of the device with respect to the user to account forchanges in position, separation, etc. One such approach that can be usedto update calibration information in accordance with various embodimentsis illustrated with respect to FIG. 6. In this example, a display page600 is illustrated that could be rendered in a browser or otherapplication on a display element of a computing device. In this example,afield of view icon 602 or other indicia is displayed on the displayelement during a calibration period when the device is unsure of thecalibration information or otherwise needs to refine the calibrationinformation. In this embodiment, the icon 602 can appear when a user'sgaze is first detected by the device. As part of the user's naturalreaction and/or as part of the instructions given to a user, the usercan attempt to look at the icon when it appears on the display element.If the icon is not in the correct position that corresponds to theuser's gaze, the user's gaze direction will move towards the icon. Insome embodiments, the icon can stay in a fixed location until the user'sgaze rests upon the icon, and the calibration information can beadjusted accordingly. In other embodiments, the user can continue toperform various actions, without having to stop and stare at the icon,and the icon can be repositioned by the system to follow the user's eyemovements. The icon can “float” with respect to the user's gazeposition, such that as the user's gaze direction moves to the right, thecalibration information can adjust as the icon nears an edge of thedisplay element or other location. For example, a substantiallytransparent icon (such that the user can see what is displayed “beneath”the icon) can appear that moves where the device thinks the user islooking. As the user moves gaze direction around the display element,particularly near edges, the icon position can adjust accordingly. Forexample, if the icon is near the right edge of the display but the userkeeps moving the gaze direction to the right, then the calibrationinformation can be updated to compensate for the difference. Once thegaze direction is calibrated sufficiently, the icon can be removed untilthe calibration procedure is determined to require recalibration.

In some embodiments, the user can turn the gaze tracking functionalityon or off, such as may be represented by a graphic 604 or other elementindicating that gaze tracking is active. In some embodiments, the usercan look at (or otherwise select) the graphic to activate the trackingicon 602 and update the calibration. As discussed, in some embodiments,the tracking icon is a semi-transparent icon that allows a user to seewhat is behind the icon while still knowing the position of the icon. Inanother embodiment, the icon is only visible (briefly) when the userperforms a quick and/or large eye movement, such as by looking up at thetop of the display, in order to communicate to the user whethercalibration is still adequate. In another example, the icon is onlydisplayed when the user is looking at a specific element, such as thetracking graphic 604, to indicate that calibration is occurring. Variousother options can be used as well within the scope of the variousembodiments.

Algorithms for determining the gaze direction of a user with respect tothe computing device can use any number of different approaches. Forexample, FIG. 7( a) illustrates an image of a face 700 of a user of adevice as could be captured (e.g. obtained, imaged) by an imagingelement of the device. Thus, the face 700 is depicted as perceived bythe imaging element of the device. As can be seen in FIG. 7( a), andalso in the eye-specific view of FIG. 7( b), there are various aspectsof the user's face that can be located and measured, such as theperceived width and height of a user's eyes, the perceived relativeseparation of a user's eyes and the perceived relative position of theuser's eyes to an edge of the user's face when facing the device. Anynumber of other such measurements or aspects can be used as should beapparent. When a user tilts or translates the device, or moves his orher head in any direction, there will be a corresponding change in atleast one of these measured aspects in subsequent images that areobtained. For example, if the user tilts his or her head right or left,the horizontal distance f in FIG. 7( a) between the user's eyes and anedge of a side of the user's face will change. In a similar manner, ifthe user tilts his or her head up or down, the vertical distance gbetween the user's eyes and an edge of the top of their head willchange. Further, the shape or horizontal measurements a and b and theshape or vertical measurements e and h of the user's eyes will changeand can change by different amounts. The separation distance c betweenthe eyes can change as well. Using such information, the device candetermine a type of motion that occurred and can use this information tohelp interpret the movement of the user's pupils or other suchinformation.

For example, FIGS. 8( a)-8(c) illustrate the movement of a user's pupilswith respect to the user's eye position. In some embodiments, the user'spupil position relative to the user's eye position can be at leastpartially indicative of the gaze direction of the user. For example,assuming the user is facing toward the device, in FIG. 8( a) the user isgazing forward, while in FIG. 8( b) the user is gazing downward and inFIG. 8( c) the user is gazing to the left (in the figure). Suchinformation by itself, however, may not be sufficient to determine gazedirection. For example, if the user had tilted his or her head up (orback) while making the pupil movement in FIG. 8( b), the user mightactually be looking forward (or even ‘up’ relative to the previousposition). Further, if the user translates his or her head to the leftor right in FIG. 8( a), but does not adjust the position of the pupilswith respect to the user's eyes, then the viewing location wouldactually change even though the user is still looking straight ahead.Thus, in certain embodiments, it can be advantageous to utilize thefacial measurement approaches discussed with respect to FIGS. 7( a)-7(b)to interpret the pupil movements of FIGS. 8( a)-8(c).

When using an imaging element of the computing device to detect motionof the device and/or user, for example, the computing device can use thebackground in the images to determine movement. For example, if a userholds the device at a fixed orientation (e.g. distance, angle, etc.) tothe user and the user changes orientation to the surroundingenvironment, analyzing an image of the user alone will not result indetecting a change in an orientation of the device. Rather, in someembodiments, the computing device can still detect movement of thedevice by recognizing the changes in the background imagery behind theuser. So, for example, if an object (e.g. a window, picture, tree, bush,building, car) moves to the left or right in the image, the device candetermine that the device has changed orientation, even though theorientation of the device with respect to the user has not changed. Inother embodiments, the device may detect that the user has moved withrespect to the device and adjust accordingly. For example, if the usertilts their head to the left or right with respect to the device, thecontent rendered on the display element may likewise tilt to keep thecontent in orientation with the user.

In some embodiments, the accuracy of the image capture and detection canbe such that gaze direction and/or field of view can be determined basedsubstantially on pupil-related information. In one embodiment, imageanalysis can be performed to locate the position of the user's pupils.The dimensions of the pupils themselves, as well as position andseparation, can be indicative of changes in the user's gazing direction.For example, in addition to determining that pupils move from left toright in adjacently-captured images, the device can determine, due tosmall changes in the width of each pupil, whether the user position withrespect to the device has translated. Similarly, the device candetermine whether the user rotated his or her eyes, which would resultin changes in diameter since the eyes are spherical and changes inrotation will result in changes in the captured dimensions. By beingable to precisely measure pupil-related dimensions, the device can trackthe field of view of the user with respect to the device.

Another benefit to being able to accurately measure pupil-relateddimensions is that the device can also determine a focus depth of theuser. For example, if the user focuses on a point “farther away” fromthe user, the device can detect a change in separation of the pupils.Because the device can also measure the dimensions of the pupils in theimage, the device can also determine that the increase was not due to anaction such as a decrease in the distance between the user and thedevice. Such information can be useful for three-dimensional images, forexample, as the device can determine not only a viewing location, butalso a depth at which the user is focusing in order to determine wherethe user is looking in three-dimensional space.

While user information such as pupil measurements can be determinedthrough various image analysis approaches discussed above, conventionalimage analysis algorithms are relatively processor-intensive and canrequire a significant amount of memory. Conventional portable devices,such as cellular phones and portable media players, might not have thenecessary resources to perform such real-time image analysis,particularly at the resolution needed to detect small variations inpupil diameter. Further, in order for the image capture to work theremust be a sufficient amount of ambient light, such that if a user isreading an electronic book on a device with a display such as anelectronic paper display that does not generate significant illuminationas would an LCD or similar display element, there might not be enoughlight to adequately capture the necessary image information.

An approach that can be used in accordance with various embodimentsinstead utilizes infrared (IR) radiation to illuminate at least aportion of the user and capture the reflected radiation. A particularadvantage of IR radiation is that the human retina acts as a reflectorwith respect to IR, such that light from a given direction will reflectback in substantially the same direction but will not reflect back in asubstantially different direction. This effect is similar to a red-eyeeffect in an image, where an image captured using a flash attached tothe camera can experience the red-eye effect, but images captured fromcameras at other angles with respect to the flash will not demonstratethe red-eye effect.

Using such an approach, a user can be illuminated with infraredradiation from the device, such as by including at least one infraredemitter in the device that will emit infrared radiation that is notharmful to the user and further cannot be detected by the user duringuse. The device also can include at least one infrared sensor fordetecting infrared radiation reflected by the user. One advantage ofsuch an approach is that the IR components can be relatively low-power,relative to illuminating a user with ambient light. Further, the imagescaptured can have a relatively low color depth, similar to a grayscaleimage, such that much less processing capacity is needed than foranalyzing full-color image analysis. In other embodiments, differenttechniques may be utilized to determine the gaze direction of a user.

FIGS. 9( a)-9(c) illustrate an example process for determining pupil orretina parameters using infrared radiation that can be used inaccordance with various embodiments. In this example, a first image isshown in FIG. 9( a) that was captured using a sensor positioned near aninfrared source, such that each retina reflects the infrared radiation.FIG. 9( b) illustrates another image captured using a sensor positionedaway from an infrared source, such that any IR radiation reflected bythe retinas is not directed to, or detected by, the sensor. Thus, as canbe seen, the major significant difference between the two images is thereflection by the retinas. Using simple image comparison or subtractionalgorithms, for example, the retinas can quickly be extracted from theimages. If noise is sufficiently filtered out, using any appropriatemethod known in the art, the resultant image in FIG. 9( c) will includesubstantially only the reflection from the retinas, which can quickly beanalyzed with very little resource allocation.

In order to accurately match the images, the images should be capturedsimultaneously or with little time between captures in order to minimizethe effect of user and/or device movement. Further, in cases where thereare two IR sensors positioned at different locations on the device, asis discussed elsewhere herein, image matching or rectifying algorithmscan be used to adjust for offsets in the images due to the differentcapture positions. Various calibration or other such processes can beused as known in the art for matching the position of items capturedfrom slightly different positions and/or angles.

As with the analysis of conventional full-color images described above,however, the resolution of the IR-based approach described above mightnot be sufficient to track gaze direction or field of view for allapplications. In such cases, it can be beneficial to utilize additionalinput mechanisms and/or additional IR emitters and detectors to helpinterpret or enhance the captured information. At least some of theseadditional elements shall be referred to herein as“environment-determining input elements,” as the additional elements areoperable to determine at least one aspect relating to the environmentsurrounding the device, such as light or noise surrounding the device, arelative orientation of the device to the surroundings, whether a useris holding the device, etc. While use of IR emitters and detectors aredescribed herein, any type of facial or movement recognition techniquemay be used with the embodiments described herein.

As an example, FIG. 10 illustrates an electronic computing device 1000that can be used in accordance with various embodiments. This exampleincludes a display element 1012 and an orientation-determining element1008, such as an accelerometer, which can be used to help interpretmotion in a captured image using various approaches described herein. Asdiscussed, the device also includes an image capture element forcapturing image information about the user of the device. While thedevice in FIG. 1 included a camera for capturing full-color images, thisexample device includes an IR emitter 1002 and two IR detectors 1004,1006 (although a single detector and two emitters could be used as wellwithin the scope of the various embodiments). In this example, a firstIR detector 1004 is positioned substantially adjacent to the IR emitter1002 such that the first IR detector will be able to capture thereflected infrared radiation from the user's retinas. The second IRdetector 1006 is positioned a distance away from the IR emitter 1002such that the detector will not detect any reflected radiation due tothe IR emitter. The emitter and detectors can be positioned on thedevice in locations that are least likely to interfere with the user'soperation of the device. For example, if it is determined that averageusers hold the device by the middle of either side of the device andprimarily on the right side or on the bottom of the device, then theemitter and detectors can he positioned at the corners of the device,primarily on the left-hand side or top of the device. In anotherembodiment, there may be additional IR emitters (not show) positioned onthe device that transmit IR at different frequencies. By detecting whichfrequencies are received by the detectors, the device can determinespecific information as to the orientation of the users gaze.

As discussed, using multiple input mechanisms can help to interpretinformation captured about the user, such as the movement of a user'spupils or other features. Further, additional input devices can allowfor the determination of actions in addition to changes in gazedirection, which can be used to control other aspects of the device aswell. These other controllable aspects can relate to conservation ofpower consumption and other such resources within the scope of thevarious embodiments.

For example, the device can include a touch-sensitive element 1010around at least a portion of the device 1000. For example, a materialsimilar to that used with a touch-sensitive display element can be usedon the back and/or sides of the device. Using such material, the deviceis able to determine whether the user is actively holding the device, inaddition to whether the user is looking at, or away from, the device.Such information could be used with the gaze information to determinehow to adjust the display element. For example, in some embodiments, thedisplay element might be configured to go into a mode with lowercontrast or brightness when the user is not looking at the displayelement but is still holding the device. The display element might onlyturn completely off if the user is not looking at the display elementand not holding the device.

In addition to determining whether the user is holding the device,through use of the touch-sensitive element, the system can determinewhat portions of the device are held/covered by the user. In such anembodiment, multiple IR emitters may be positioned on the device atdifferent locations, and based on where the user is holding the device(i.e., which IR emitters are covered vs. not covered), the system candetermine which IR emitters to use when capturing images.

The ability to determine such user information can also be used toadjust functionality of other elements of the device as well. Forexample, the device can include a number of input buttons or keys (notshown) that can be used for purposes such as to input characters,activate voice dialing, dial emergency numbers or perform any of anumber of other such functions. If a user is not holding the device, atleast some of these inputs and/or actions might be disabled in order toprevent the user from inadvertently activating one of those functions,such as by accidentally pressing one of the buttons when placing thedevice in a bag or pocket, dropping the device, etc. Further, in deviceswhere the keypad is illuminated, the keypad lighting can be turned offwhen the device is not being held by the user and can be turned back onwhen the user picks up the device, which can be detected by acombination of the touch-sensitive element and orientation-determiningelement(s), for example.

The example device in FIG. 10 also includes a light-detecting element1016 that is able to determine whether the device is exposed to ambientlight or is in relative or complete darkness. Such an element can bebeneficial in a number of ways. In certain conventional devices, alight-detecting element is used to determine when a user is holding acell phone up to the user's face (causing the light-detecting element tobe substantially shielded from the ambient light), which can trigger anaction such as the display element of the phone to temporarily shut off(since the user cannot see the display element while holding the deviceto the user's ear). The light-detecting element could be used inconjunction with information from other elements to adjust thefunctionality of the device. For example, if the device is unable todetect a user's view location and a user is not holding the device butthe device is exposed to ambient light, the device might determine thatit has likely been set down by the user and might turn off the displayelement and disable certain functionality. If the device is unable todetect a user's view location, a user is not holding the device and thedevice is further not exposed to ambient light, the device mightdetermine that the device has been placed in a bag or other compartmentthat is likely inaccessible to the user and thus might turn off ordisable additional features that might otherwise have been available. Insome embodiments, a user must either be looking at the device, holdingthe device or have the device out in the light in order to activatecertain functionality of the device. In other embodiments, the devicemay include a display element that can operate in different modes, suchas reflective (for bright situations) and emissive (for darksituations). Based on the detected light, the device may change modes.

Further, a light-detecting sensor can help the device compensate forlarge adjustments in light or brightness, which can cause a user'spupils to dilate, etc. For example, when a user is operating a device ina dark room and someone turns on the light, the diameters of the user'spupils will change. As with the example above, if the device includes adisplay element that can operate in different modes, the device may alsoswitch modes based on changes in the user's pupil dilation. In order forthe device to not improperly interpret a change in separation betweenthe device and user, the light detecting sensor might cause gazetracking to be temporarily disabled until the user's eyes settle and arecalibration process is executed. Various other such approaches tocompensate for light variations can be used as well within the scope ofthe various embodiments.

The device may also include a distance or range detector that determinesthe distance between the device and the user. When a change in distancebeyond a predefined threshold is determined, the font of the device maybe altered or the display element shut off. For example, if the distancebetween the user and the device increases beyond a specified threshold,the rendered font size may automatically increase or be rendered with ahigher contrast to make viewing easier. Likewise, if the distancebetween the device and the user decreases, the font may automaticallydecrease in size. If however, the distance between the device and theuser exceeds a second threshold (e.g., the user is no longer detected),the device or the display element may shut off.

The example device 1000 in FIG. 10 is shown to also include a microphone1014 or other such audio-capturing device. The device in at least someembodiments can also determine various actions based upon sound detectedby the microphone. For example, the device might be able to determinefrom the detected sound whether the microphone is up against a material,which might cause sound to be muted or otherwise distorted. If thedevice is in a pocket or bag, for example, the microphone might besignificantly covered by a material, which can affect the quality ofsound recorded. In some embodiments, the device might emit (periodicallyor continually) a sound that is not perceptible to humans, but that canbe detected by the device. In this way, the device can determine whetherit is likely in a location that cannot readily be accessed by a user,such that certain functionality can be disabled, In some embodiments,the sound might only be emitted when the device is attempting todetermine a present location of the device. In some embodiments, thesound emitted might be perceptible to a human, but would only be emittedwhen the device is attempting to determine whether to shut down certainfunctionality. In some embodiments, the device might emit (and record) anoise when a certain button is pushed, for example, to determine whethera user likely pushed the button or whether the button was likelyaccidentally pressed by something in the surrounding container, etc.

Using the microphone, the device can disable other features for reasonssubstantially unrelated to power savings. For example, the device canuse voice recognition to determine people near the device, such aschildren, and can disable or enable features, such as Internet access orparental controls, based thereon. Further, the device can analyzerecorded noise to attempt to determine an environment, such as whetherthe device is in a car or on a plane, and that determination can help todecide which features to enable/disable or which actions are taken basedupon other inputs. If voice recognition is used, words can be used asinput, either directly spoken to the device or indirectly as picked upthrough conversation. For example, if the device determines that it isin a car, facing the user and detects a word such as “hungry” or “eat,”then the device might turn on the display element and displayinformation for nearby restaurants, etc. A user can have the option ofturning off voice recording and conversation monitoring for privacy andother such purposes.

There can be any of a number of functions performed based upon any of anumber of user actions detected with respect to a device. As a simpleexample, FIG. 11 illustrates a table 1100 of situations that can eachhave specific actions taken for a single device. In this example,different actions can be taken based on whether a user is determined tobe looking at the display element of a device (referred to in FIG. 11 as“DE”), looking near the display element or device, looking substantiallyaway from the device or looking away from the device for an extendedperiod of time. For each of these situations, there can be differentactions taken that are based at least in part upon input from otherelements of the device. For example, if the user is holding the devicebut the user is looking away from the device, the display element mightturn off but all other input functionality might remain active. If thedevice is facing up but at rest, the device might have different actionsspecified than if the device is moving randomly or is in the dark. Asshould be understood, the table can have many more dimensions, as therecould be specific actions taken for combinations of more than two ofthese and other such situations.

Further, the action taken can depend at least in part upon anapplication or other function executing on the device. For example, if auser is playing/displaying a video file on the device, then the displayelement might not shut off if the user is not holding the device orsitting directly in front of the device, such that the user's gazedirection is detected. If the user is only listening to audio, however,the device might power down the display element and disable otherfunctions if the user is not looking at the display element or holdingthe device. If the user has a word processing application open, but isnot holding or looking at the device, many functions of the device mightpower down until the user again interacts with the device. Various othersituations and actions are possible within the scope of the variousembodiments.

In some embodiments, these functions can be at least partiallyconfigurable by a user of the device. For example, FIG. 12 illustratesan example of a configuration display element 1200 that could bedisplayed to a user of the computing device. In this example, the useris able to enable/disable functions, such as gaze-based display elementactivation 1206 and interaction-based function control. For example, theuser can be presented with several options for each of a number ofsituations 1202. In this example, the user has selected that the displayelement brightness should lower when the user glances away from thedisplay element and turn off when the user looks away from the device.There can be any of a number of actions for each situation and any of anumber of different situations specified. There can be other inputs aswell, such as a sensitivity adjustment 1204 that can adjust how quicklythe device reacts or the thresholds used to control device actions. Inone embodiment, a highly sensitive setting can cause the device displayelement to shut off as soon as the user looks away from the displayelement, while a low sensitivity setting might cause the device to lowerbrightness of the display element, or power off, only when the userlooks away for a certain amount of time or looks to a further distancefrom the device, etc.

FIG. 13 illustrates an example process 1300 that can utilize settingssuch as those set forth in FIG. 12 to perform actions in response tovarious situations. In this example, gaze tracking is activated 1302automatically or in response to user input. Once activated, the devicecan continually monitor user gaze direction, or field of view, as wellas device state information 1304. As discussed above, device stateinformation can include any type of information input or determined fromany of a number of sources. For example, state information can includewhether the device is being held by a user, such as may be determined bya touch-sensitive element, as well as whether the device is out in theopen or in a contained environment, as may be determined by alight-detecting element, microphone circuit or other such element. Thestate information can also include information pertaining toapplications or processes currently executing on the device.

The device can determine whether the region (e.g., at or away from thedevice) in which the user is gazing changes or if the state of thedevice changes in a significant way (e.g., the user is no longer holdingthe device) 1306. If either of these changes occur in a meaningful way.the device can look up the appropriate action to take for the change(s)1308 as may be contained in memory, in a remote data store, in usersettings or any other appropriate location. Once the appropriate actionis determined, that action can be performed 1310. As discussed, this caninclude any of a number of actions, such as activating or deactivating adisplay element or enabling/disabling functionality associated withvarious device inputs, etc. The device can continue monitoring todetermine subsequent changes that require additional actions.

In many of the above examples, the actions taken by the device relate todeactivating certain functionality for purposes of reducing powerconsumption. It should be understood, however, that actions cancorrespond to other functions that can adjust similar and otherpotential issues with use of the device. For example, certain functions,such as requesting Web page content, searching for content on a harddrive and opening various applications, can take a certain amount oftime to complete. For devices with limited resources, or that have heavyusage, a number of such operations occurring at the same time can causethe device to slow down or even lock up, which can lead toinefficiencies, degrade the user experience and potentially use morepower.

In order to address at least some of these and other such issues,approaches in accordance with various embodiments can also utilizeinformation such as user gaze direction to activate resources that arelikely to be used in order to spread out the need for processingcapacity, memory space and other such resources. For example, FIG. 14illustrates an example display 1400 including a plurality of icons, eachof which has an action that can potentially be started before the useractually selects that icon. For example, a user wanting to play a movieon a DVD drive might look at the DVD drive icon (as shown by the viewlocation graphic 1404) up to a couple of seconds before the user moves acursor 1402 to that icon or otherwise selects that application. In someembodiments, when the device detects that the user is looking at the DVDdrive icon, the device can begin spinning the DVD such that when theuser accesses the DVD software, the disk will be spinning at the properspeed, and the device can quickly locate the desired information. Insome embodiments, the device might also begin loading the DVD softwareinto memory or otherwise perform operations that will reduce the timeneeded to launch the software. Similarly, a user looking at a disk driveicon or other such interface element can cause the appropriate componentor element to prepare for access.

In some embodiments, a user might look at an icon for a Web browser or ahyperlink in a document, which can cause the device to submit a requestfor content to a home page or other URL designated to be opened with thebrowser application or hyperlink, such that the time needed to loadand/or render the page upon execution of the browser application will bereduced. If a user looks at an icon for an e-book reader application oremail program, for example, the device can begin loading content orpreparing resources that will be needed for those applications. If auser looks at an icon to power down the device, for example, the devicecan begin to close down any unnecessary applications or otherwiseprepare to shut down the device. Even if the user does not actuallypower off the device, shutting down unnecessary applications can reducepower and resource consumption, etc. In still another example, thedevice may detect when the user is reaching the last few words in a pageand automatically initiate a page turn, without requiring the user tophysically interact with the device. Additionally, if a video oranimated ad is presented on the display element, the device may onlyplay or animate the content when the user positions their gaze over thatcontent.

Further, the device can conserve resources by changing the way in whichthe device functions for specific applications. For example, FIG. 15illustrates an example display 1500 wherein product information for ahierarchical categorization is displayed in a three-dimensionalinterface. When a device determines the viewing location and/or focuslocation of the user, the device can determine the category level of theinterface that is currently being viewed by the user. In this situation,the device can conserve resources by only rendering the current level,and/or adjacent levels, at the full resolution, color depth, etc. Forexample, in the display 1500, it is determined that the user is lookingat the consumer products level 1502. The device can then render thelevel at full resolution, color depth, etc., which can include pullinghigh resolution images for products at that level to be displayed to theuser. Information at lower levels, such as at the ‘computers’ and ‘PC’levels 1504 in this example, can be rendered at a lower resolution,etc., which can conserve processor usage, memory and power. Further, theimages displayed at the non-current levels can be lower resolutionimages, or even stock or default images, that simply indicate that itemsare contained within those levels. As should be apparent, various otherapproaches to rendering information at different levels can be used aswell.

While the amount of power and resources saved when viewing a categoricalinterface on a cell phone may he minimal, there are applications anddevices where the savings in power and/or resources can be significant.For example, FIG. 16 illustrates an example of a display 1600 for adevice that renders graphical information in three-dimensions, as above,but in a processor-intensive environment such as a video game orthree-dimensional map that is capable of being viewed on a large formatdisplay device, such as a widescreen television. The device in this casemight include multiple connected components, such as a computing deviceand separate display device, but the device can still determine usergaze direction with respect to the display. If a user is viewinginformation on a large format display, there will be a portion of theregion that is in the direct viewing region of the user, and there canbe other portions that are in the peripheral view of the user. Devicesin accordance with certain embodiments can conserve processing power, ordirect processing power where it is needed most, by treating the directand peripheral view of the user differently.

For example, the computing device could determine (using one of theaforementioned or other calibration procedures) where the user islooking on the display element, as well as a boundary 1602 defining aprimary view region 1604, based upon factors such as determined distancefrom the display element, display size, etc. The primary view region canbe centered about a current viewing location of the user with respect tothe display. Any items or content viewable within the primary viewregion can be rendered at full resolution, and items or content outsidethe primary viewing region can be rendered at a lower resolution, if atall. For example, the illustration shows a sidewalk visible in theprimary viewing region. If there is a gold coin on the sidewalk outsideof the viewing region, and the coin is an artistic feature that does nothave any effect on the game, map, etc., then the device might not renderthe coin unless the user's gaze adjusts such that the viewable area nowincludes the coin, at which point the coin could be rendered. In otherembodiments, the coin could be rendered whenever the coin is displayedon the display element but might only be rendered in full resolutionwhen the viewing region includes the coin.

As discussed, the device can also determine a focus depth of the user.If the interface displayed is truly a three-dimensional interface (asopposed to a two-dimensional representation of a three-dimensionalinterface), then the device can selectively render items in the viewableregion based upon a focus region or focus plane of the user. Forexample, there is a person in a window of a building in the viewingregion in FIG. 16. Unless the user is focused on that building in thedistance, the device would not have to render that person at fullresolution. As discussed, such an approach can not only conserveprocessing power but can help to direct available processing power whereit is needed most such that a device can appear to have more processingcapability than is actually present. This approach also provides abenefit to users in that they are not overloaded with information whilemaintaining a sense of the whole rendered environment. In some devices,the device might actually fully render the entire image much of the timebut can prioritize portions of the rendering such that differentrendering levels can be used when needed. For example, in a game where aplayer is walking alone through an alley, the device might render theentire alley. If a swarm of zombies enters the alley, however, aspectsof the alley might be rendered at a lower rendering level in order toprovide the processing capability needed to render the additionalcharacters. In addition to rendering portions of the objects at a lowerrendering level, the user may also be able to directly interact withobjects in the user's primary focus region. Continuing with the zombieexample, the user could zap or otherwise destroy the zombies by lookingdirectly at them. In such an example, the system detects that the useris looking directly at the zombie and, based, for example, onpredetermined game behavior, detects that when the user is lookingdirectly at certain objects in the game (the zombies), certain actionsare taken.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 17 illustrates an example of an environment 1700 for implementingaspects in accordance with various embodiments. As will be appreciated,although a Web-based environment is used for purposes of explanation,different environments may be used, as appropriate, to implement variousembodiments. The environment 1700 shown includes both a testing or adevelopment portion (or side) and a production portion. The productionportion includes an electronic client device 1702, which can include anyappropriate device operable to send and receive requests, messages orinformation over an appropriate network 1704 and convey information backto a user of the device. Examples of such client devices includepersonal computers, cell phones, handheld messaging devices, laptopcomputers, set-top boxes, personal data assistants, electronic bookreaders and the like. The network can include any appropriate network,including an intranet, the Internet, a cellular network, a local areanetwork or any other such network or combination thereof. Componentsused for such a system can depend at least in part upon the type ofnetwork and/or environment selected. Protocols and components forcommunicating via such a network are well known and will not bediscussed herein in detail. Communication over the network can beenabled by wired or wireless connections and combinations thereof. Inthis example, the network includes the Internet, as the environmentincludes a Web server 1706 for receiving requests and serving content inresponse thereto, although for other networks an alternative deviceserving a similar purpose could be used, as would be apparent to one ofordinary skill in the art.

The illustrative environment includes at least one application server1708 and a data store 1710. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein the term “data store” refers to any device orcombination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server can includeany appropriate hardware and software for integrating with the datastore as needed to execute aspects of one or more applications for theclient device and handling a majority of the data access and businesslogic for an application. The application server provides access controlservices in cooperation with the data store and is able to generatecontent such as text, graphics, audio and/or video to be transferred tothe user, which may be served to the user by the Web server in the formof HTML, XML or another appropriate structured language in this example.The handling of all requests and responses, as well as the delivery ofcontent between the client device 1702 and the application server 1708,can be handled by the Web server. It should be understood that the Weband application servers are not required and are merely examplecomponents, as structured code discussed herein can be executed on anyappropriate device or host machine as discussed elsewhere herein.Further, the environment can be architected in such a way that a testautomation framework can be provided as a service to which a user orapplication can subscribe. A test automation framework can be providedas an implementation of any of the various testing patterns discussedherein, although various other implementations can be used as well, asdiscussed or suggested herein.

The environment also includes a development and/or testing side, whichincludes a user device 1718 allowing a user such as a developer, dataadministrator or tester to access the system. The user device 1718 canbe any appropriate device or machine, such as is described above withrespect to the client device 1702. The environment also includes adevelopment server 1720, which functions similar to the applicationserver 1708 but typically runs code during development and testingbefore the code is deployed and executed on the production side and isaccessible to outside users, for example. In some embodiments, anapplication server can function as a development server, and separateproduction and testing storage may not he used.

The data store 1710 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing production data 1712 and user information 1716,which can be used to serve content for the production side. The datastore also is shown to include a mechanism for storing testing data1714, which can be used with the user information for the testing side.It should be understood that there can be many other aspects that mayneed to be stored in the data store, such as page image information andaccess rights information, which can be stored in any of the abovelisted mechanisms as appropriate or in additional mechanisms in the datastore 1710. The data store 1710 is operable, through logic associatedtherewith, to receive instructions from the application server 1708 ordevelopment server 1720 and obtain, update or otherwise process data inresponse thereto. In one example, a user might submit a search requestfor a certain type of item. In this case, the data store might accessthe user information to verify the identity of the user and can accessthe catalog detail information to obtain information about items of thattype. The information can then be returned to the user, such as in aresults listing on a Web page that the user is able to view via abrowser on the user device 1702. Information for a particular item ofinterest can be viewed in a dedicated page or window of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include computer-readablemedium storing instructions that, when executed by a processor of theserver, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 17. Thus, the depiction of the system 1700 in FIG.17 should be taken as being illustrative in nature and not limiting tothe scope of the disclosure.

The various embodiments can be further implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would he familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CCI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or or any scripting language, such as Perk Python or TCL,as well as combinations thereof. The server(s) may also include databaseservers, including without limitation those commercially available fromOracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary tiles for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch-sensitive displayelement or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (“RAM”) orread-only memory (“ROM”), as well as removable media devices, memorycards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices or any other medium which canbe used to store the desired information and which can be accessed by asystem device. Based on the disclosure and teachings provided herein, aperson of ordinary skill in the art will appreciate other ways and/ormethods to implement the various embodiments.

The specification and drawings are, accordingly, to he regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

1. A computer-implemented method of operating a computing device,comprising: obtaining image data including a representation of at leastone portion of a user using a camera of the computing device; analyzingthe image data to determine a position of at least one feature of theuser relative to a region of a display element of the computing device;and utilizing the position to adjust a power consumption rate of thecomputing device.
 2. The computer-implemented method of claim 1, furthercomprising: adjusting the region after a specified period.
 3. Thecomputer-implemented method of claim 1, wherein utilizing the positionto adjust the power consumption rate includes: decreasing the powerconsumption rate of the display element based at least in part upon theposition corresponding to the user gazing away from the region.
 4. Thecomputer-implemented method of claim 3, wherein utilizing the positionto adjust the power consumption rate includes: increasing the powerconsumption rate of the display element based at least in part upon theposition corresponding to the user gazing toward the region.
 5. Thecomputer-implemented method of claim 1, further comprising: deactivatingfunctionality of at least one input element of the computing devicebased at least in part upon the position corresponding to the usergazing away from the region.
 6. The computer-implemented method of claim5, further comprising: activating the functionality of the at least oneinput element based at least in part upon the position corresponding tothe user gazing toward the region.
 7. The computer-implemented method ofclaim 5, wherein the at least one input element comprises one of atouch-sensitive element, a keyboard, a keypad, a roller button, a secondimaging element, an audio-capturing element, a power button, a powercharging element, or a physical switch.
 8. The computer-implementedmethod of claim 1, further comprising: obtaining environmental data fromat least one environment-determining element of the computing device,wherein the power consumption rate is further adjusted based at least inpart upon the environmental data.
 9. The computer-implemented method ofclaim 8, further comprising: adjusting functionality of at least oneinput element of the computing device based at least in part upon theposition and the environmental data.
 10. The computer-implemented methodof claim 8, wherein the at least one environment-determining elementcomprises one of a touch-sensitive element, an orientation determiningelement, an audio recording element, or a light sensing element.
 11. Thecomputer-implemented method of claim 1, further comprising: animating anelement displayed on a display element based at least in part upon theposition corresponding to the user gazing toward the region.
 12. Acomputing device, comprising: a processor; a display element; a camera;and memory including instructions that, upon being executed by theprocessor, cause the computing device to: obtain image data including arepresentation of at least one portion of a user using the camera;analyze the image data to determine a position of at least one featureof the user relative to a region of the display element; and utilize theposition to adjust a power consumption rate of the computing device. 13.The computing device of claim 12, wherein the instructions, upon beingexecuted, further cause the computing device to: adjust the region aftera specified period.
 14. The computing device of claim 13, wherein theinstructions to cause the computing device to adjust the region afterthe specified period include causing the computing device to: decrease asize of the region.
 15. The computing device of claim 12, wherein theinstructions, upon being executed, further cause the computing deviceto: adjust functionality of at least one application executing on thecomputing device based at least in part upon the position.
 16. Thecomputing device of claim 12, wherein the instructions, upon beingexecuted, further cause the computing device to: render at least oneelement displayed on a display element of the computing device in afirst manner based at least in part upon the position corresponding tothe user gazing away from the region; and render the at least oneelement in a second manner based at least in part upon the positioncorresponding to the user gazing toward the region.
 17. The computingdevice of claim 12, wherein the instructions, upon being executed,further cause the computing device to: adjust at least one of acontrast, a brightness, or a resolution of at least one elementdisplayed on the display element based at least in part upon theposition.
 18. The computing device of claim 12, wherein theinstructions, upon being executed, further cause the computing deviceto: utilize calibration information to analyze the image data.
 19. Anon-transitory computer-readable storage medium storing instructionsthat, upon being executed by a processor of a computing device, causethe computing device to: obtain image data including a representation ofat least one portion of a user using a camera of the computing device;analyze the image data to determine a position of at least one featureof the user relative to a region of a display element of the computingdevice; and utilize the position to adjust a power consumption rate ofthe computing device.
 20. The non-transitory computer-readable storagemedium of claim 19, wherein the instructions, upon being executed,further cause the computing device to: adjust the region after aspecified period.
 21. The non-transitory computer-readable storagemedium of claim 19, wherein the instructions, upon being executed,further cause the computing device to: obtain environmental data from atleast one environment-determining element of the computing device,wherein the power consumption rate of the display element is furtherbased at least in part upon the environmental data.
 22. Thenon-transitory computer-readable storage medium of claim 21, wherein theinstructions, upon being executed, further cause the computing deviceto: adjust functionality of at least one input element of the computingdevice based at least in part upon the position and the environmentaldata.
 23. The non-transitory computer-readable storage medium of claim19, wherein the instructions, upon being executed, further cause thecomputing device to perform at least one of: adjust functionality of atleast one application executing on the computing device based at leastin part upon the position.